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Helmet System, for Informadon or Wespon Systems 



g |PT.n A>JD BACKGROUN r> THE INVENTION 

Hie piesent invention relates to helmets for use in automated systems and. in 
particular, it concerns a helmet system for use with weapon or information systrans which 
requires minimal integration with other systems. Various aspects of the invention relate to a 
helmet position tracking system and an eye-motion tracking system, and the associated 
meOiods of operation. 

It has become incieaangly common for automated systems, particularly in the field of 
aeronautics, to employ systems integrated with a hehnet worn by a pilot as an integral part of 
an automated system. For example, in hehnet sights, a helmet position sensing system follows 
the angular position of the hehnet and directs a weapon system to aHgn with a fixed sight 
mounted on the hehnet In more sophisticated systems, a hehnet mounted display (HMD) 
provides numerous additional features, including providing visible indicators aUgned with 

olgects viewed by the pilot 

In all such systems, the position (angular position and/or linear displacement) of the 
helmet relative to the platform on which it is used must be measured to a high degree of 
accumcy. On fixed terrestrial platforms, hehnet position monitoring may be performed using 
rdatively simple and low-cost inertial sensors, alone or in combination with other sensors. For 
moving platforms, however, hehnet-mounted inertial sensors are not sufficient due to flie non- 
inertial 0 6., subject to acceleration) nature of the platform itself. Thus, pilot hehnet position 
systons for use m aircraft generally employ either a magnetic or^ an optical position 
measoranent system. 

Magnetic hehnet portion sensing syst^ are widely used, but suffer ftom a number 
of disadvantages. Most notably, the use of magnetic hehnet position sensmg systems has a 
highly labor intenavo set-up procedure, requiring time-consuming mapping of tiie magnetic 
fields of the entire codqnit envhonment, and requirmg re-mappmg whenever a change is made 
to tiie codcpit arrangement 

Optical hehnet position sensing systems, on tiie other hand, suffer firom tiieur own 
disadvantages. Optical systems typically «npIoy a coclq»it-mounted imaging sensor to 
identify optical markers sndi as active LED's or reflective patches located on die hehnet If 
tiie hehnet can turn timmgh a wide range of angiles, tiie optical maricers may not always be 
wifhm tiie field of view CTOVO of flie miagfa»g sensor. Where rehable continnous hehnet 
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tracking over a wide range of angles is required, multiple image sensors viewing from 
different angles may be needed. 

US. Patent No. 6,377»401 to Barflett desKibes a hybrid system in ^ch a helmet- 
moimted camera obtains images of active maikers located in the cocIq>it as a self-dieck or 
5 correction for measurements by a magnetic sensor system. 

All of the aforementioned types of hehnet position sensors require a significant degree 
of integration into tiie airraaft systems. Specifically, components of flie magnetic and/or 
optical ^^stem must typically be installed in various locations witiiin the cockpit Furtiiennore, 
the systems typically require transfer of data via, the aircraft electronics systrans, or 

1 0 alternatively, via dedicated installed wiring. In either case, die process of integration requires 
re-evaluation and testing for the safety, operational and reliability standards required by the 
relevant aviation auflioiities, a process which is typically very cosfly and may take momhs or 
years. These testing and certification procedures themselves act as a major deterrent to 
adoption of many new systems which, in themselves, wonld otherwise be highly 

15 advantageous. 

A co-assigned, co-pending U.S. Patent Application, published as Publication No- 
20020039073 discloses a helmet-based cuing system which employs eye-tracking to provide a 
wide range of advanced features without requiring a hehnet mounted display. This document 
is hereby incorporated by refermce as if fuUy set out herein. 

20 Although offering many advantages, flie aforementioned patent application desaribes a 

system based upon otherwise conventional hehnet position sensing systems and eye-gaze 
direction sensing systems. As a result, the implementation of each subsystem inherentiy 
requires some significant degree of integration into the cocApit environment 

Turning now to eye-tracking systems, it is known to use images of the eye together 

25 with image processing to derive the gaze direction of the eye. Commercial eye-tracking 
systems are available fix)m ASL Applied Science Laboratories (Bedford, MA, USA) and &om 
SR Researdi Ltd. (MEississaug^, Ontario, Canada). These systems typically operate usmg IR 
viwelengdi illumination and imaging of tiie eye in order to avoid the visual distiffbance wfaidi 
would be caused by illumination with visible light 

30 Existing eye-tracking systems generally operate in one or other of two modes. In a first 

mode, the syston id^tifies the position of the pxqpil and of a direct corneal reflection or 
'*gline' of tiie reflected illumination source. The gaze direction is then derived &om the vector 
diffeence b^een the p\xpi\ oentroid and the glint This mode can provide good results which 
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.„ r.,a«vdy to«»sWv. to vibration or rm»Ugmn=* of tte ,pa.«»s. Howov«. 
pU^ mode is ordy op«.tiv. ov« 3 ,el«ivel, »aU rang, of angle »h« ft. <tao« 
L«I .rfl«ttoo U visible to .he imaging sensor. For applications ^ this «nan range of 
™,„ is insufficient, a difEnent mode relying upon pupU position «Jy is .Bed. Tta p^- 
5 only mode is bi^y sendtive to nnsalignment of 4e apparatns and oth« m.d«nucal 
disturbances. As a tesult, no currently available system is capable of tracking eye-movements 
ov« a ™de range of angles vMe also ooas>ensating fct errors dn. to shiOng of dip^ 

and otbw mechanical disturbances. 

Ttere is therefore a need for a hebnet system aad corresponding methods for use m 
10 automated systems ^^ch would provide an accurate indication of hebnet podtion and/or eye- 
^ direction of a user on a stationary or moving platform while minimizing the reqmred 
of integration into existing systems of the platform. It ^vould also be highly 
advantag^us to provide an eye-tracking system and conesponding me^ .«hich wonld track 
eye movements over a wide range of angles while providing automatic correction for 
1 S vaiiatioDs in Systran aligmnrat 
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CTTM MARY OF THR TNVENTION 

The present invention provides a hebnet position measuring system and a hebnet 
mounted eye-gaze direction sensing system, together with associated methods. 

According to the teachings of the present mvention there is provided, a hebnet position 
n^easuiing system for use m a predefined «ivixomnent, the system comprising: (a) a hebnet- 
mounted iUumination system for directing electromagnetic radiation of at least one 
wavelength from the hebnet to at least one range of angles; (b) a set of at least three passive 
reflectors deployed at fixed positions in the predefined environment so as to reflect 
electromagnetic radiation from the iUumination system; (c) a hebnet-mounted imagmg system 
sensitive to at least the at least one wavelength for derivmg images of part of the predefined 
environment including electromagnetic radiation reflected fi^m the reflectors; and (d)a 
p««essing Systran associated wifli the imaging system for processing the images to identify 
regions of Ae images corresponding to die reflectors and hence to determine mformation 
30 relatmg to a poation of the hebnet within the predefined environment- 
According to a fiirtha feature of the present mvention. the illumination system 
includes at least one infrared LED. 
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According to a further feature of the present invendon, Ae imaging system is at least 
partially selective to electromagnetic radiation of at least one wavelength- 
According to a fiirther feature of the present invention, the illumination sy stem directs 
the eleclromagnetic radiafion substantially continuously wiftin a horizontal ang^lBr rang^ of 
5 at least 60^ 

According to a further feature of the present invention, the illumination system directs 
fixe electromagnetic radiation substantially continuously within a vertical angular range of at 
least 40°, 

According to a further feature of the pres^ invention, at least part of the processmg 
1 0 system is located in a housing external to, and electrically interconnected wifli, the helmet, the 
housing being confined for wearing on the body of a user. 

According to a ftatiier feature of the present ur^ention, fliete is also provided an 
inertial measurement system associated with the hehnct and connected to the processing 
system for providing additional information relating to a position of the helmet 
15 According to a fiirther feature of the present mvention, fhe inertial measuremrat 

system includes three angular motion ^aisors deployed m fixed relation to ttie helmet so as to 
sense rolational motion about three orthogonal axes. 

According to a further feature of the present mvention^ the helmet has a convexly 
curved external surface, and wherein the three angular motion sensors are mounted in 
20 proximity to substantiaDy mutually orthogonal regions of the curved external surface. 

According to a further feature of the present invention, the helmet has a convexly 
curved external surface, the system further comprising a covct element attached to the hehnet, 
fhe cover el^ient having a concave surface feeing the convexly curved external surfece of the 
helmet, wherein the three angular motion sensors are mounted relative to the covet element at 
25 substantially mutually orthogonal regions of the concave surface. 

According to a furft^ feature of the present invention, the predefined enviromnent is 
part of a moving platform, the moving platform having at least one associated platform 
position measuremrat system^ the hehnet position measuring system further comprising a 
communications link associated wifli the processing system and with at least one element on 
30 the moving platform, the communication link transfering platform position information 
derived from the at least one platform position measuremait systCTi to the processing system, 
and wherein tiie processing system is configured to compute inertially-derived relative motion 
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irf<,™.«on to mcrton of to l»ta« within fte predefined envixomnen. by con>p.«ng 

4e infbm«ti«. ftom <1» inert.1 .neasurcmen. system «ft d« platfonn position infonn-»n. 
According to a fcrtoer feanne of 4e present invention, the processing syst«n » 
h, employ «. mi^tive filter calculation to combine the InetiaBy-derived rdativ. 
5 motion inft»m=don«rfte position in«»ma.ion derived ftom the in^«>gener.«>^^ 

helmet position infonnalion- v 

Accoiding to a further feature of &e present invention, the commumcaHons brfc is 
implemettledasavrirelesscoiiimumcationslink. 

According to a ftarther featuic of the present invention, the communications link is 
10 «iociatedv«thatleasto^eoftheg^o^^,:aprocessing^^ 

unit within a DMSsilelaiindiea^. 

According to a fiinherfteturootftepesont invention, there is also provided a hetaet- 
monnlrf eywraidng sjtton traddng a gazo direction of at least one eye relative to ±e 

helmet 

,5 According to a Setter featnre of the p««« invention, the eye-tracktag system IS 

fl» processh^ system, the processing system calculating a gaze direction of 

tiie at least <me eye retative to «» predefined environment 

there is also fMvided ««ading to the teachings of the present invention, a hehnet 
^ito measuring syston fl» d««mi.ing tire portion of a hehnet relative to a movmg 
20 platfom.ti» moving pl«8»m having an toertial navigation system, tite system 

Man inertial measmonent system associ«ed v*h fte hehnet; (b)a commumcanon hnk 
.«ccUn«l vnth hod. fl« hehnet and to ptatJ^ ti.e commmncation link transf^ 
4„m Are inertial navig-ion system to hdme.: and (c) a processing system associated ««h 

ft. m«sm«nen. system and 4. commnmcation link. fl» processing system 

25 pcoc.ssh«d.tatomti»h.«rti.lmo«»nam««systmandti.edauftomtiaei^^ 

to derive inertidJy-demred hdmat position d-a indicative of ti>e hohnel pos,t,on 

leladve to flie moving platfonn* 

According to a furflrer feature of the present invention, the processing system is 
configured to perfonn transfer aHgmnent of the inertial measurement system firom the mertial 
30 navigation system of the platfonn. 

Acconfing to a finther feature of the present invention, the inertial measurement 
system includes three angular motion s«isors deployed in fixed relation to the hehnet so as to 
sense rotational motion about three orthogonal axes. 
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According to a furtiicr feature of flie present invention, liie helmet has a convexly 
curved external sur&ce, and wherein the three angular motion sensors are mounted in 
proximity to substantially mutually orthogonal regions of the curved external suifece. 

According to a further feature of the present invention, the helmet has a convexly 
5 curved external surface, the syst^ ilirth^ comprising a cover element attached to the helmet, 
the cover element having a concave surfece feeing the convexly curved external surfece of the 
helmet, ^^^rein the three angular motion sensors are mounted relative to the cover element at 
substantially mutually orthogonal regions of &e concave surface. 

According to a furdier feature of the present invention^ there is also provided an 
10 optical measuring system associated >vith the processing system, the optical measuring system 
including: (a) at least three markers mounted on a first of the helmet and the moving platform; 
(b) at least one camera mounted on the oth^ of the helmet and &e moving platform for 
genCTating an image of at least the mark^; and (c) image processing means for processing 
the image to generate optically-derived helmet position data, wherein the processing system is 
15 additionally for co-processing the inertially-derived helmet position data and the optically- 
derived helmet position data to generate overall helmet position information. 

According to a further feature of ti» present invention, the camera is mounted on the 
helmet, and wherein the at least three markers are mounted on the moving platform. 

According to a fur&er feature of the ptesent invention, the optical measuring system 
20 includes at least one illumination source mounted on the helmet, and wherein the at least three 
mariasrs are passive reflective markers* 

According to a finrther feature of the present invention, there is also provided a helmet- 
mounted eye-tracking syst^ for tracking a gaze direction of at least one eye relative to the 
l^lmet 

25 According to a further feature of the fniesent invention, the eye-tracking system is 

associated with the processing systenoi, tiie processing system calculating a gaze direction of 
the at least one eye relative to the moving platform. 

There is also px>vided according to &e teachings of the present invention, a hehn^ 
assembly having a position measuring Systran, die helmet assembly comprising: (a) a helmet 

30 having a convexly curved external surfece; and (b) an in^al measurement system including 
fbxee angular motion sensors deployed in fixed relation to the hdmet so as to sense rotational 
motion about three orthogonal axes, v^rein the three angular motion sensors are mounted in 
proximity to substantially mxztually orAogonal regions of the curved external surfece. 
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There is also provided according to the teachings of the present invention, a hehnet 
ass«nbly having a position measuring system, the helmet assembly comprising: (a) a hehnet 
having a convexly curved external surface; (b)a cover element attached to the hehnet. the 
cover el^nent having a concave surfece facing the convexly curved ext^ surfece of the 
5 helmet and (c) an inertial measurement system including three angular motion sensors for 
sensing rotational motion about three orthogonal axes, wherein the three angular motion 
sensors are mounted relative to the cover element at substantiaUy mutually orthogonal regions 

of the concave surfece. 

There is also provided according to the teachmgs of tbe present hivaition. a method 

10 for reliable real-time calculation of pupU gaze direction over a ^de range of angles, flie 
method comprising: (a) iUuminating an eye .vith electromagnetic radiation of al least one 
wavelength; (b) obtaining an image of the illummated eye; (c) identifying within the image a 
pupil location; (d) automaticaUy detennining whether the image includes a direct corneal 
reflection; (e) if the image does not include a direct corneal leflection. calculating a current 

15 pupa gaze direction based upon the pupU location, the calculating being performed ushig a 
pupil-only g92e direction model; (f) if iniage does include a direct corneal reflection, 
deriving a current pupU gaze direction based upon bo* tiie pupH location and a position of ti»e 

direct corneal reflection. 

According to a furtiier feature of the present invention, at least one parameter of tiw 
20 pupil-only model is updated based upon at least one pupil gaze direction derived from both 
tiie piqnl location and tiie position of direct corneal reflection. 

pp yPT? nR.SCRlPTIO >J OP THE DRAWINGS 

The invention is herem described, by way of example only, witii reference to tije 

25 accompanying drawings, vdierein: 

no. 1 is a block diagram of a hehnet system and related components, constructed and 
operative according to tiie teachings of tiie present invention, tiie hehnet system including 
inertial motion sensors, an optical position sensor arrangement and eye-tiracking sensors; 

no. 2 is a sdiematic representation of a preferred implementation of an inertial, or 
30 initial-optical hybrid, hehnet position subsystem, constructed and operative according to the 
teachmgs of the present invention, from tiie system of Figure 1 ; 

no. 3A is a schematic view of an hnplranentation of tiie hehnet system of Figure 1 ; 
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FIG. 3B is a schematic representation of a prefetred geomeby of layout fox the mertial 
sensors of the hehnet system of Figure 1 associated with a curved sur&ce of a hehnet; 

FIG. 4 is a schematic representation of a prefeired implementation of the optical 
position sensor arrangCTQent of the helmet system of Figure 1; 
S FIG. 5 is a flew diagram illustrating the operation of the optical position sensor 

arrangement of the hehnet system of Figure 1; 

FIG. 6 is a schematic &ont view showing a j«:eferred implementation of an eye 
tracking seosox of flie helmet system of Figure 1; 

FIG* 7 is a schematic plan view of the eye tracking sensor of Figiure 6; 
10 FIG. 8 is a photographic representation of an eye showing the pupil centroid and the 

direct corneal reflection of an illumination source; 

FIGS. 9A-9C are schematic representations Illustrating the effects of eye motion on 
piq>il position and direct corneal reflection; and 

FIG. 10 is a flow diagram illustrating a prefeired mode of operation and corresponding 
15 method for deriving eye g^ze direction according to the teachings of the present invention. 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

The presait invention provides a helmet position measuring system and a helmet 
moimted eye-^2:e direction sensing system, together with associated methods. 
20 The principles and operation of systrais and methods according to the present 

invention may be better understood with reference to the drawings and tiie accompanying 
description. 

Referring now to the drawings. Figure I shows a helmet system, generally designated 
10, constructed and operative according to the teachings of the present invration, together 

25 with a nmnber of related components. In general terms, the preferred embodiment of helmet 
system 10 shown here includes a number of subsystems each of ^^ch has utility in itself 
when used together with various otherwise conventional systems, but which are synergiously 
combined in the prefmred embodiment as will be described. These subsystems include a 
hehnet tracking system based upon one, or preferably both, of an inertial sensor system or 

30 inertial measurement unit C*IMU") 12 and an optical smsor arrangement 14, and an eye- 
tracking system 16a, 16b for tracking movraaent of one^ or preferably both, eyes of a user. 
Each of these subsystems and its particular novel features wUl be described separately below. 
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T«o common eonsWenttions p«rvade prrfmed taplemeataflons of *e ™ious 
„a«yst«o, of ti»p«s«t mvenlioD. Rfitly. each s,*«ysKm fa pief«r>bly impkmoDttd 

««««« inleg^tton inU. MccBoxdc systems of to pWfo^. or a. leas, nto^g m 
ttegr-ion « ftr « possible, as ^ be de^Ued bdow. TOs gready snopUfles the 
5 i„stdMo»procetoa»a 6cmta.es ••i.lroar of to systems on existing 

to am. levd of evaluaflon and testfng a, ,™uld be ■e.piied an tau^ 

The second consideratiou pervading prefen«d implementations of fte vanous 
subsystems of tbe present inverrticm is the desire to minimize the excess weight and btUk of 
10 the hehn« so that the hehnet remains as close as possible to the size and weight of a 
conventional "dnmb" helmet To Ibis en4 any components ^ch do not need to be hehnet- 
„,ounted are p^feably mounted in a separate body-mounted unit 18 (Figure 3 A) which ^ 
^xn or otherwise strapped to ^ body of the user. Ibis subdivision of componaxts is 
represented schematically in Figure 1 by dashed Une A-A with components above the hne 
15 being hehnet-mounted and conqKments below the line being body-mounted. Tbus. m most 
preferred implementations, the total weight of all of the helmet-mounted electromc 
components of the system is no more to about 300 grams, and preferably no more than 200 
Furthermore, most preferred unpl«nentations of the helmet maintam the generally 
spherical outer sheU shape of the hdmet standing no more than about 6 cm. and preferably no 
20 moxethanabout4cm,fiomtheheadoftheuseroversubstantiaUyaUofitssurfece.Tberesd^ 

is a hehnct which feels snnilar to a standard hehnet and greatly reduces the physical stress on 
the user compared to exisliag hi-tedi hehnet systems. 

la a further related conaderation, the safety of the user is prefembly enhanced by use 
of low power electronic components so as to avoid hi^-power connections between the 
25 hehnet and platfonn systems. Accordmg to one option, a power supply 19 may be a self- 
contamed battery unit, thereby avoidmg power-supply connection to tixe platform. More 
preferably, a dmple power-jadc comiector is used to sapply low-voltage power to the hehnet 
system. A batt«y power s,^ 19 may optionally be used to back-up tbe external power 
connectiozL 

30 The various subsystems of preferred implementations of the invention wJl now be 

described mdividually. 
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Tni»-rriftl Uftlm et Position Subsystem 

As mentioned eariier, helmet-mounted inertial trackir^ systems alone are insufiBcient 
to deteraiine motion of a helmet relative to a non-inertial platfomi- To address this problem, 
the inertial tracking system of tiie present invention preferably provides an inertial 
5 measurement system v^ch includes an inertial measurement unit 12 associated wifli the 
helmet, and a communication Hrik (transceivCTS 10a and 206) associated with both the helmet 
and fhe platform for conveying data from an inertial navigation system ("ENS") 500 of the 
platform to the helmet system* A processing system 22 associated wiHi inertial measurement 
unit 12 and communication link 22ff, processes data from inertial measurement xmit 12 and 

10 from the inertial navigation system 500 to derive helmet position data indicative of the helmet 
position relative to the moving platform. Parenthetically, it shotild be noted that the term 
"helmet position*' when used herein as a stand-alone term is used to refer to either or both of 
angular position (attitude) and linear spatial position (displacement). When referring to 
parameters of motion, the convention of *^oa[tion**, "velocity" and "attitude** is used v/herein 

15 "position** refers specifically to position in three-dimensional ^ace relative to a set of 
T&Sbtmce coordinates. 

In addition to die basic functionality of calculating differential motion, it is a ftirther 
feature of particularly preferred implementations of the present invention that tratxsfer 
alignment is used to "align** the reference axes of IMU 12 with tije reference axes of INS 500, 

20 thereby enhancing the precision of the measurement, bringing the ou^ut of the small and 
relatively low-precision head-mounted system up to a precision close to that of tibe much more 
sophisticated platform INS. Transfer alignment is a well known technique, typically used for 
inertial measurement systems rigidly fixed, or at least tethered, to a common platform, for 
correcting one system on the basis of a more accurate system moving on tihe common 

25 platform. Transfer alignment has not heretofore been CTtiployed in a helmet tracking system 
and would convrationally be discounted as impossible since tiie helmet is essentially free to 
move with the head of the user relative to the platfonnu In practice, however, for a rapidly 
moving platform such as an aircraft, the present invention points out that the velocity of the 
helmet may be £issumed for csdculational purposes to be idratical to that of the platform. 

30 Based vxpoa this observation, the present invCTtion teadies the use of transfer alignment for 
enhancing the predsion of measurement A furth^ distinctive feature of preferred 
implementations of the transfer alignment of the presoit invention is tfiat flie moving platform 
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INS motion data for performing the transfer alignment is transmitted to Ae helmet system 
wiielessly via the ^vireless communications link (transceivers 20a and 20*). 

A preferred implementation of the inertiaJ. or hybrid, helmet position subsystian is 
aiustrated scbemalicaUy in Figure 2. He basic inertial hehnet position calculation employs 
angular rate sensor inputs from a set of gyros at 200 and linear acceleration sensor ii^nits from 
a set of acceleiometers at 202 which are processed by a strap-do^vn processing module 204 of 
processing system 22. Strap-down processing module 204 employs standard inertial sensor 
integration techniques well known in the art to detemoine &e motion parameters G)Osition 
206 velocity 208, attitude 210) of the hehnet relative to a given frame of refct«ice. refeiredto 
as ^-level local-North" (abbreviated to "LLLhT)- system also inputs at 212 &e 
platform motion data from mS 500 for platform attitude 214. velocity 216 and position 218 
relative to the given reference frame (LLLN). Hehnet attitude 210 and platform attitude 214 
are then co-p«>cessed at 220 to derive the motion, particularly the angular position or 
•^tode", of the hehnet relative to the platform, referred to herein as the "di^ential hehnet 
motion". This differential hehnet motion is tiie output of the hehnet tracking subsystem and is 
generated continuously at a refresh rate coirespondmg to tiie availabihty of the IMO and INS 
data, typicaUy m the range of 50-100 Hz. Altiiough illustrated here as deriving only die 
attitude of die hehnet. which is typically tiie only motion data which is significant for 
detemtining dhections to objects distant from the user, it will be clear that otiier motion 
parameters such as position or velocity can readily be retrieved by similar comparison of die 
corresponding ou^uts of strap-down processor 204 and tiie platform INS data. 

m addition to die basic calculation module described tiius fer, die hehnet motion data 
for velocity 208 and attitude 210. and tiie platform motion data for attitude 214. velocity 216 
and position 218 are preferably fed to Kahnan filter 222 which unplsnents transfer aligmnent 
25 algoritimis to generate corrections to mcrease accuracy of tiie inertial measurement unit 
ou^ Preferably, tiie corre«jticns mclude sensor corrections 224« and 224* for correcting 
bias or otiier enors in flie readings from the inertial sensors, and velocity and attitude 
corrections 226 whidi adjust tiie current output motion data parameters which also serve as 
the basis fiir tiie subsequent integrated motion data calculations. The implementation of tiie 
transfer aligmnent filter is essentially flie same as is used conventionally in many "smart" 
weapon systems, and will not be discussed here in detail. 

The collections 224«. 224* and 226 are typically updated at a rate limited primarily by 
tiie piocessmg capabiHties or by tiie cjuantity of data required for efective convergence of tiie 
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transfer aiigoment calculatioDS. A typical example for application of these corrections would 

be a rate of about 1 Hz. 

As will be discussed further below, flie helmet tracking system is preferably 
implemrated as a hybrid system whidi includes additional hehnet traddng subsystems, and 
5 most preferably, an optical helmet tracking system 14. In this case, Kalman filter 222 provides 
a Hgbly effective tool for combimEqg the available infonnadon from multiple sources, with 
differing refresh rates, and with self-adaptive relative weighting of the information soim^. In 
the case of an optical subsystem which measures helmet attitude relative to the platfomi, a 
preprocessing step is performed by filter 222 to transform the measurements by use of 
1 0 platfomi attitude data 214 mto the LLLN frame withm vAich the Kahnan filter computation is 
performed- 

As mrationed earlier, most preferred implementations of the present invention try to 
T^infmiyft integration of the sxxbsystans with the platform electronics systems, hi order to 
obtain the required data from the INS in a minimally integrated way, the commimication link 

15 22b is prefaubly a wireless coimnunication link associated with a peripheral device which 
already has read-acces to the INS data. In the preferred example illustrated here, 
communication link 22b is associated with a weapon interface and controller 24 which 
interfeces with a weapon system 502. Weapon system 502 is itself cormected to a data bus 504 
or equivalent dedicated wiring which makes avaUable information from multiple systems of 

20 die platform, including from INS 500. Thus, weapon interface and controller 24 can access 
data from INS 500 without itself being direcfly integrated in the electronics systems of the 
platforiTL 

By way of a practical example, in the case that weapon system 502 is an advanced 
missile system including one or more missile having its own internal INS, a data btas 

25 connection providmg the missile system with ain^aft INS data typically already exists in 
order to allow transfer alignment of the missile INS using the aircraft data as a reference. In 
this case, by tapping into the missile data directly, the data required by helmet system 10 may 
be retrieved without any modification of the aircraft hardware or software. The data 
connection may be achieved ^ther tbrou^ connection with a processing unit within the 

30 missile itself, or tiirough coimection with a processing imit within the missile laimcher unit 

Turning now additionaUy to Figures 3 A and 3B, the helmet-moimted IMU 12 typically 
has sets of linear and rotational motion sensors which need to be mounted in mutually 
orthogonal geometric relation. Specifically, the IMU typically includes three rotational rate 

12 
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sensors denoted "A", "B" and "C", and three linear acceleiometers denoted «X", and «Z" 
(Figure 1). As mentioned earUer. it is a particular feature of most preferred implanartations 
of the present invention that tiie hehnet system maintains a low profile api«0)dmating to a 
conventional hehnet profile. To this end, the present invention prefaably makes use of the 
inherent curvatm^ of the hebnet surface to locate a set of three sensors where Ihey can be 
mounted paiaUel to the local surfece and still be mntuaUy orthogonal to the other two sensors. 
In practice, tins is typically achieved as shown in Figure 3 A by providing a cov« element 26. 
similar to a standard visor cover, ri^dly attadied to the helmet 28. Cover elanent 26 is 
formed with a concave surfece feeing the conespondmg convexly curved e>rtemal surfece of 
hehnet 28. By suitable positioning of tiie angular motion saisors "A". «B" and "C" (and/or 
linear motion sensors «X% "Y" and rZ"). it can be ensured that they are mounted relative to 
the cover element at substantially mutuaUy orthogonal regions of the concave surfece. For 
clarity of presentation, cover element 26 is shown here to be transparent to reveal the 
underlying components. Alternatively, the components may be mounted dneotly under, or 
15 over, the convexly curved external surface of the hehnet itself to achieve an equivalent 
^liietrical arrangement. Figure 3B is a schematic representation Dlustrating one posable 
choice of positions on a convexly (or concavely) curved surfece which provide mutually 

or&ogonal mounting positions. 

Although the inertial hehnet position sensing system described thus fer is beUeved to 
be highly effective in its own right, most preferred implementations of the present mv«ition 
employ a hybrid hehnet trackmg system with a second preferably optical subsystem providing 
corrective data. A preferred example of the optical hehnet position subsystem will now be 
desMibed wititi reference to Figures 1 , 2 and 4. 
nptirat Helmet Pf>gitj9P Snbsvstan 
25 In order to provide an optical hehnet position tracking system with minimal 

integration into systems of the platform, it is a particularly feature of most prefenred 
implanenlations of tbe optical trackmg system that the only "instaUed" elements outside the 
hehnet system itself are passive reflectors 30, typically appUed as stickers positioned within 
the cockpit or other working environment At least three, and typicaUy four, reflectors 30 are 
30 used, and Ihey may have identical shapes and sizes, or may be geometrically distinct. The 
reflectors are preferably directional reflectors which reflect maxhnum mtensity along a line 
roughly parallel wifli the incoming illumination. 



20 
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In order to operate with pasave reflectors 50» (^cal ssdsot aiiangement 14 includes a 
helmet-mouBted illuminadon system 32 for diiectiag electromagnetic radiation of at least one 
wavelength fiom the helmet in at least one range of angles, and a hedmet-mounted imaging 
system 34 sensitive to at least the at least one wavelength for deriving inwges of part of tbe 
predefined environment including electromagnetic radiation reflected from reflectors 30. 
Processing system 22 then processes the unages to ideaH^ regions of the images 
corresponding to reflectors 30 and hence to detamine infonnation relating to a position of 
helm^ 28 within tbe predefined envixonmCTl 

Preferably, iUumination system 32 includes at least one mfiraied LED, and most 
preferably two, three or four LED's which together cover substantially the entire field of view 
of imaging system 34. This preferably corresponds to a substantially contimious hori2»ntal 
angular range of at least 60", and a substantially continuous vertical angular range of at least 
45». In this context, the terms "horizontal" and "vertical" are used to refer to directions as 
perceived by the user in his or her normal orieaatation on tiie plat&rm. Optionally, the optical 
system may be supplemented by one or more additional illumination system 32 and ima^^g 
system 34 mounted on the helmet with additional viewing directions in order to enlarge the 
range of angles over which reflectors 30 are witiiin tiie FO V. Alternatively, an enlarged set of 
reflectors may be positioned to provide distinctive reflective symbols over an increased range 
of angles and/or in different viewing directions. For example, a secondary set of IR reflective 
stickeis which are transparent to visible li^ may be d^loyed on a codqpit canopy to provide 
optical tracking vfhsa the user looks "up" in an aircraft fiame of reference. 

For reliable optical tracking, it is desired to achieve high contrast imapng of reflectors 
30 while usmg low power illumination, despite die feet that the system operates m an 
environment vMch may be exposed to direct solar radiation. Surprisuagly, it has been found 
that ttiese conditions can be met very successfully by employing directional reflectors 0.e., 
vMch retum a majority of the reflected illumination intensity in a direction roughly parallel 
with the incoming illumination) in combination witii narrow waveband wavelaigtii selection. 
Thus, in most preferred implementations, at least the imaging system 34 is configured to be at 
least partiaUy selective to electromagnetic radiation of a wavelengtii or wavelength band 
emitted by illumination system 32. This can be achieved most simply by positioning a suitable 
filter elemaot 36 in front of at least the ima gi n g sensor 34. 
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^ «Ubr«i=n procedure ^ f"^^ ^ *=<"°^°" 

ft^ fl„ obUin«l ar. too™ in to art a«. arc Wicdly dndU. » *o» of ft. 

^onaneidaBy available systems mentioned earUer. 

a^a™^ Tie ine^lal system ofi«s large Wwidft (-pid tesponse) and <^ ov^ 
^:l^^.«l aaH- ranse. but may su*r *>m e^ts o. "dHfl". P=«.^ 

condiHons «bcre insufflcien. data may be available for effe^ve ^ 
Uom^ The opdoal system on U.e other hand, once ca^btated. r.p»^e 

drift but suffer. 60m reUttvely slow response (typically around 5 Hz) and U-ed 
.^mnge.Il«tw„^*ereforecompl«nen.eachoftcrperfeo.y«.provKteah^ 

traCdng system vWch combines the advantages of both snbsystems. A pr^ 
amctnre huegtaflng the m«»u.emen«, of fte diffirent subsystems 

with referOTc© to Figure 2. 

Hgure 5 .bo»s a preferred se^ of ope^on of the optical helmet posttron 
ttselt 11» optical sensor subsyst^n first obtains opfical h^ 
and prods the images .0 chec. whether ^cient maricers 3. a.. v«hm the 
JJm^ of View (Step 4»). If it^cien. makers are included in the -P'«^ "^ » 
new image is sampled (renmr to step 46). When su«den. marUe. are moluded m fte fields 

View, the image is then processed .0 derive fte hehne. position relative ^^^^^^ 
Ms helmet position data is then output at step 52 to d» Kahmm filter 222 (Ftgure 2) 

,a«e it is comltod v«.h «» ofl« available d.«. « pmvide optoal overaU accuracy. 

Tp..«-T«»>irfTifr Subsystem ^^^wH 

T»ntognow*>*e ^r^traddng subsystem and associated meftod. a 
„^ tayout of fte ey^.r»*ing optical components is ilhtstnted in Kgures 6 and 7 He 
„„^,s are essemially similar to ftose of con.««ional eye-tracMng ^.^V.^ 
^ mmnination ,yst«n (I£D «iO) and an mftared imaging sensor (camenr 62) deployed, 
^psetivdy. fcr iltaminating and hnagin, eye of the user. The geometrical "^^--''^ 
^ hl^ve. to minimis obsouratton of fte user's field of view -1 .0 ~ 
^„,,h.c«mponen.swi.«n.h.c«,vendon.lhelmetprom.Tottusen^ oth^^^ 

and camera 62 prrf-^ably vl«v the eye via a "hot mir«»- 64 mounted m iron, of to eye. 
^ on to intend surfece of a visor. The .«m "ho. mirror" is used hetein .0 reform^ 

^ILa. etanea. which i. reflective » to mlevan. fiequendes of IR radrahon «hde havmg 
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high transparency to optical wavelengths of light In order to minimize the interference of 
outside light sources (including the sun) on measurements, the visor itself may 
advantageously be designed to exclude the relevant j&equencies of IR radiation. In the case of 
an anti-laser visor for excluding certain wavelCTigflis of incoming laser radiation, the already 
5 existing filtraed wavelengths can be used to advantage by the eye tracking system. 
Alternatively, illimdnation and imaging may be perfoimed in solar-blind frequency bands 
where ambirat radiation levels are very low. An example of the resulting image is shown in 
Figure 8 vAisi& the pitpil region is clearly identifiable as the dadcest region 100 and ihe glint is 
the brightest spot 102. 

1 0 The \ise of hot-minor 64 enables LED 60 and camera 62 to be located in the peripheral 

region of helmet 28 near the edge of the visor. For extra compactness, depending upon the 
size and shape of camera 62, it may be advantageous to raiploy an extra mirror 66 to allow 
motmting of the camera vertically or in any other preferred orientation. 

The eye-tracking subsystem also include processing and data storage components, as 

15 well as power supply and driva: circxdtry, as will be clear to one ordinarily skilled in the art 
The processing and data storage components are typically included in the general designation 
of processing system 22 (Figure 1) and may be implemented as dedicated components within 
that system, or shared components which additionally serve other subsystems. 

Turning now to the operation of the eye-tracking subsystem and the corresponding 

20 mediod, as mentioned earlier, fliere are two known techniques for drnving eye-gaze direction 
from inoages of the eye, referred to herein as "pupil-plus-glinf ' and "piq)il only". These 
individual techniques are known per se and are included in commercially available products 
as detailed in the backgroxmd to the invention above. Figures 9A-9C illustrate a range of eye 
positions. In Figures 9A and 9B, botii the pupil region 100 and the gjint 102 are clearly 

25 visible. This allov^ use of the pupil-plus-glint gaze direction derivation which offers high 
precision and stability, and rejects helmet movements etc hi Figure 9C however, the corneal 
reflection is lost due to the hi^ an^e of the eye relative to the illimnnation and images 
direction. According to the teachings of the prior art, if measurements are required at such 
high angles, the entire system would need to work m a pupil-only mode, with a consequ^ 

30 loss of precision and stability. For op^stional s^plicadons, this lack of stability could render 
the entire syst^ ineffective. 

To address this problem, it is a particularly prefCTred feature of the eye-tracking 
subsystem and corresponding mediod of the present invention that it combines the stability of 

16 
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the pupU-plus-glint tracking method vnth a range of tracking angles beyond 4e langp which 
provides direct corneal reflection. This is achieved by vsms real-time automatic switchang 
between two tracking calculation techniques, and most preferably, by automatic self- 
calibration of the pupil-only tracking technique based upon output of the pupil-plus-gbnt 
calculation tedmique during continuous operation of the system. 

Turning specificaUy to Figure 10. a method according to the piesait invention for 
xeliable real-time calculation of pupil gaze direction over a ^de range of angles obtains an 
image of the illuminated eye (step 70). preferably via the apparatus of Figures 6 and 7. The 
system then processes the image to identify tiae pupil and. if avaflable, tbe corneal reflection 
or "glint" (step tl). Ihese can be identified readily by threshold techniques alone, or m 
combination with other shape and/or position based algorithms. A. centroid of tiie piq>il 
position is then calculated (step 74). typically by best fit of an ellipse to the pupil region. At 
step 76 the system automatically determines whetiier the image includes a direct corneal 
reflection. If it does, the system proceeds at step 78 to calculate the vector between the ghnt 
centmid and the pupil centroid and to calculate the gaze direction based t,.on this vector (step 
80) If the "glintf is not avaUable. a gaze direction calculation is made at step 82 using a 
pupil-only gaze direction modeL Ihis "model" may be represented in any suitable form 
including, but not Hmited to. an algebraic formula and a look-up table or values. 

According to most preferred miplementations of the present mvention, values of the 
gaze direction derived fiom the pupil-plus-glint calculation are used to update at least one 
parameter of tiie pupU-only model (step 84). In die case of an algebraic formula, this is 
typically done by adjusting one or more coefficient.of the fomiula. to the case of a look-up 
table, adjustment may be made either to individual values or by scaling a plurality of values. 

By updating the pupil-only model ftequentiy. or substantially continuously, it can be 
ensured tiiat tbe pui«l-only model is optimized for the current position of flie hehnet and 
>vorking conditions, tiiereby substantially eliminating the cumulative sources of error 
normally associated with the piqnl-only eye-tracking technique. 

According to a farther supplementary, or alternative, aspect of tiie present invention, it 
is possible to i«ovide multiple Dhimmation directipns of the eye such that at least one direct 
corneal ^t is recdved by camera 62 over an enlarged range of gaze direction angles. TTie 
additional iUuminalion directions are most simply achieved by providing additional hot- 
xniirors 64 suitably angled and positioned across tiie hmer surfece of the visor, eac* witii its 
own illumination source (LED 60). Although it is possible to use multiple cameras for eacli 
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eye to achieve a siinflar result, the use of a single camera vntix multiple iUumination directioiis 
is typically prefened for its reduced image procesang load. The m a tch i ng of eaidb glint wilh 
the corresponding iilmninadoii diiectioa is typically straightforward by use of the relative 
geometry of the pupil and glint positions in ihe images. According to one prtferred altemadve 
5 implementation, a total of Aree or more illumination directions are used to rasure a direct 
glint over substantially the entire range of angular motion of the eye, thereby rMdering die 
use of die pig)il'Only mode uimecessary. 

In die preferred case of binocular eye tracking, the two uidividual eye-gaze directions 
are correlated at st^ 86. In die case of assumed fiur-vision parallel binocular fixation^ the two 
10 individual gaze directions may be assumed to be parallel and can be combined to iinprove 
output accuracy. Each measuranent may be given equal weight, or an adaptive filtear 
technique may be used to give variable weight depending upon differmt xeg^ons of greater or 
lesser measurement accuracy for each eye, or as a function of which calculation tedmique was 
used for each eye. 

15 In the case of the preferred combination of features of the present invention, the eye- 

gaze direction relative to die hehnet is then combined widi helmet position data ixq>ut at step 
88 and the gaze-direction relative to the platfomi is calculated (step 90). 
Additional Options 

The hehnet system described herein is useful for a wide range of different 
20 ^plications. In the specific version shown herein in the drawings, it is particularly useful as 
part of a system such as is described in the aforem^itioned co-assigned, co-pending U.S. 
Patent Application, published as Publication No. 20020039073 to provide a hehnet-based 
cuing syst^ without requiring a hehnet mounted display. It should be noted, however, that 
any or all of the features of die present invention may equally be used to advantage in the 
25 context of a helmet which includes a helmet mounted display (HMD). 

It should also be noted that the helmet system of the present invention, with or without 
a HMD, may also be used as a pow«iul tool for training or debriefing users. Specifically, it 
will be noted diat prefraed implementations of systrai 10 mherently graierate helmet tracking 
information, eye tracking information and a forward-looking image fir^m image system 34. By 
30 recording some or all of this Hnfa, optionally time-correlated to other actions of the user, 
databuis information or external evCTts, it is possible to reconstruct the movements of the 
user's head and his or her eye motion in the context of the forward view image. The data may 
either be recorded witldn data storage devices witiiin processing system 22 or by a separate 



18 



PCT/IL2004/001067 

WO 2005/052718 

data storage unit (not shown) with a hard-^viIed or vSieless one-dixectional co«uBunications 
link. The data storage device may optionaUy be part of aa impact-protected disaster- 

inv^gation system. ^ 

By v.ay of one non-limiting example, it is possible to replay the forward-view video 
images with the user's gaze direction sxq,erimposed thereon, thereby documenting the visual 
awareness of the user and the time^iivision of his or her attention. Optionally, the playback 
mode can simultaneously display fli^t information of the aircrafl. as well as fhgjrt 
information of other aircraft or any other data or parameters available fiom the databus. The 
combined data can also be used to reconstruct progression of events in three-^Jimensions. 

It wiU be appreciated that the above descriptions are intended only to serve as 
examples, and that many other embodiments are possible within the s«^ of the present 
invwition as defined in the app^ed claims. 
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